Object classification for vehicle radar systems

ABSTRACT

Methods and systems are provided for object classification for a radar system of a vehicle. The radar system includes a transmitter that transmits radar signals and a receiver that receives return radar signals after the transmitted radar signals are deflected from an object proximate the vehicle. A processor is coupled the receiver, and is configured to: obtain spectrogram data from a plurality of spectrograms pertaining to the object based on the received radar signals; aggregate the spectrogram data from each of the plurality of spectrograms into a computer vision model; and classify the object based on the aggregation of the spectrogram data from each of the plurality of spectrograms into the computer vision model.

TECHNICAL FIELD

The present disclosure generally relates to vehicles, and moreparticularly relates to methods and radar systems object classificationfor vehicles.

BACKGROUND

Certain vehicles today utilize radar systems. For example, certainvehicles utilize radar systems to detect and classify other vehicles,pedestrians, or other objects on a road in which the vehicle istravelling. Radar systems may be used in this manner, for example, inimplementing automatic braking systems, adaptive cruise control, andavoidance features, among other vehicle features. While radar systemsare generally useful for such vehicle features, in certain situationsexisting radar systems may have certain limitations.

Accordingly, it is desirable to provide techniques for radar systemperformance in vehicles, for example that may be used to provideimproved classification of objects. It is also desirable to providemethods, systems, and vehicles utilizing such techniques. Furthermore,other desirable features and characteristics of the present inventionwill be apparent from the subsequent detailed description and theappended claims, taken in conjunction with the accompanying drawings andthe foregoing technical field and background.

SUMMARY

In accordance with an exemplary embodiment, a method is provided forobject classification for a radar system of a vehicle. The methodcomprises obtaining spectrogram data from a plurality of spectrogramspertaining to an object proximate the vehicle based on received radarsignals for the radar system, aggregating the spectrogram data from eachof the plurality of spectrograms into a computer vision model, andclassifying the object based on the aggregation of the spectrogram datafrom each of the plurality of spectrograms into the computer visionmodel.

In accordance with an exemplary embodiment, a radar control system isprovided. The radar control system comprises a transmitter, a receiver,and a processor. The transmitter is configured to transmit radarsignals. The receiver is configured to receive return radar signalsafter the transmitted radar signals are deflected from an objectproximate the vehicle. The processor is coupled the receiver, and isconfigured to obtain spectrogram data from a plurality of spectrogramspertaining to the object based on the received radar signals, aggregatethe spectrogram data from each of the plurality of spectrograms into acomputer vision model, and classify the object based on the aggregationof the spectrogram data from each of the plurality of spectrograms intothe computer vision model.

DESCRIPTION OF THE DRAWINGS

The present disclosure will hereinafter be described in conjunction withthe following drawing figures, wherein like numerals denote likeelements, and wherein:

FIG. 1 is a functional block diagram of a vehicle having a controlsystem, including a radar system, in accordance with an exemplaryembodiment;

FIG. 2 is a functional block diagram of the control system of thevehicle of FIG. 1, including the radar system, in accordance with anexemplary embodiment;

FIG. 3 is a functional block diagram of a transmission channel and areceiving channel of the radar system of FIGS. 1 and 2, in accordancewith an exemplary embodiment;

FIGS. 4A-4C are flowcharts of a method for classifying objects for aradar system, which can be used in connection with the vehicle of FIG.1, the control system of FIGS. 1 and 2, and the radar system of FIGS.1-3, in accordance with an exemplary embodiment;

FIG. 5 is a flowchart of a sub-method of the method of FIGS. 4A-4C,namely, of utilizing a computer vision model for classification, inaccordance with an exemplary embodiment in which clustering is used aspart of the computer vision model;

FIG. 6 provides a flow diagram for the sub-method of FIG. 5, inaccordance with an exemplary embodiment;

FIG. 7 is a flowchart of a sub-method of the method of FIGS. 4A-4C,namely, of utilizing a computer vision model for classification, inaccordance with another exemplary embodiment in which sums of values areused as part of the computer vision model; and

FIG. 8 provides a flow diagram for the sub-method of FIG. 7, inaccordance with an exemplary embodiment.

DETAILED DESCRIPTION

The following detailed description is merely exemplary in nature and isnot intended to limit the disclosure or the application and usesthereof. Furthermore, there is no intention to be bound by any theorypresented in the preceding background or the following detaileddescription. As used herein, the term module refers to any hardware,software, firmware, electronic control component, processing logic,and/or processor device, individually or in any combination, includingwithout limitation: application specific integrated circuit (ASIC), anelectronic circuit, a processor (shared, dedicated, or group) and memorythat executes one or more software or firmware programs, a combinationallogic circuit, and/or other suitable components that provide thedescribed functionality.

FIG. 1 provides a functional block diagram of vehicle 10, in accordancewith an exemplary embodiment. As described in further detail greaterbelow, the vehicle 10 includes a radar control system 12 having a radarsystem 103 and a controller 104 that classifies objects using radarsignal spectrogram data in combination with one or more computer visionmodels.

In the depicted embodiment, the vehicle 10 also includes a chassis 112,a body 114, four wheels 116, an electronic control system 118, asteering system 150, and a braking system 160. The body 114 is arrangedon the chassis 112 and substantially encloses the other components ofthe vehicle 10. The body 114 and the chassis 112 may jointly form aframe. The wheels 116 are each rotationally coupled to the chassis 112near a respective corner of the body 114.

In the exemplary embodiment illustrated in FIG. 1, the vehicle 10includes an actuator assembly 120. The actuator assembly 120 includes atleast one propulsion system 129 mounted on the chassis 112 that drivesthe wheels 116. In the depicted embodiment, the actuator assembly 120includes an engine 130. In one embodiment, the engine 130 comprises acombustion engine. In other embodiments, the actuator assembly 120 mayinclude one or more other types of engines and/or motors, such as anelectric motor/generator, instead of or in addition to the combustionengine.

Still referring to FIG. 1, the engine 130 is coupled to at least some ofthe wheels 116 through one or more drive shafts 134. In someembodiments, the engine 130 is also mechanically coupled to atransmission. In other embodiments, the engine 130 may instead becoupled to a generator used to power an electric motor that ismechanically coupled to a transmission.

The steering system 150 is mounted on the chassis 112, and controlssteering of the wheels 116. The steering system 150 includes a steeringwheel and a steering column (not depicted). The steering wheel receivesinputs from a driver of the vehicle 10. The steering column results indesired steering angles for the wheels 116 via the drive shafts 134based on the inputs from the driver.

The braking system 160 is mounted on the chassis 112, and providesbraking for the vehicle 10. The braking system 160 receives inputs fromthe driver via a brake pedal (not depicted), and provides appropriatebraking via brake units (also not depicted). The driver also providesinputs via an accelerator pedal (not depicted) as to a desired speed oracceleration of the vehicle 10, as well as various other inputs forvarious vehicle devices and/or systems, such as one or more vehicleradios, other entertainment or infotainment systems, environmentalcontrol systems, lightning units, navigation systems, and the like (notdepicted in FIG. 1).

Also as depicted in FIG. 1, in certain embodiments the vehicle 10 mayalso include a telematics system 170. In one such embodiment thetelematics system 170 is an onboard device that provides a variety ofservices through communication with a call center (not depicted) remotefrom the vehicle 10. In various embodiments the telematics system mayinclude, among other features, various non-depicted features such as anelectronic processing device, one or more types of electronic memory, acellular chipset/component, a wireless modem, a dual mode antenna, and anavigation unit containing a GPS chipset/component. In certainembodiments, certain of such components may be included in thecontroller 104, for example as discussed further below in connectionwith FIG. 2. The telematics system 170 may provide various servicesincluding: turn-by-turn directions and other navigation-related servicesprovided in conjunction with the GPS chipset/component, airbagdeployment notification and other emergency or roadsideassistance-related services provided in connection with various sensorsand/or sensor interface modules located throughout the vehicle, and/orinfotainment-related services such as music, internet web pages, movies,television programs, videogames, and/or other content.

The radar control system 12 is mounted on the chassis 112. As mentionedabove, the radar control system 12 classifies objects using radar signalspectrogram data in combination with one or more computer vision models.In one example, the radar control system 12 provides these functions inaccordance with the method 400 described further below in connectionwith FIGS. 4A-4C, and the various embodiments also described furtherbelow in connection with FIGS. 5-8.

While the radar control system 12, the radar system 103, and thecontroller 104 are depicted as being part of the same system, it will beappreciated that in certain embodiments these features may comprise twoor more systems. In addition, in various embodiments the radar controlsystem 12 may comprise all or part of, and/or may be coupled to, variousother vehicle devices and systems, such as, among others, the actuatorassembly 120, and/or the electronic control system 118.

With reference to FIG. 2, a functional block diagram is provided for theradar control system 12 of FIG. 1, in accordance with an exemplaryembodiment. As noted above, the radar control system 12 includes theradar system 103 and the controller 104 of FIG. 1.

As depicted in FIG. 2, the radar system 103 includes one or moretransmitters 220, one or more receivers 222, a memory 224, and aprocessing unit 226. While the transmitters 220 and receivers 222 arereferred to herein as the transmitter 220 and the receiver 222, it willbe appreciated that in certain embodiments multiple transmitters 220and/or receivers 222 may be utilized, as indicated above. Thetransmitter 220 transmits radar signals for the radar system 103. Thetransmitted radar signals collectively form a beam that is transmittedby the radar system 103 for detecting objects (e.g. other vehicles,pedestrians, trees, rocks, debris, road characteristics, and so on).After the transmitted radar signals contact one or more objects on ornear a road on which the vehicle 10 is travelling and isreflected/redirected toward the radar system 103, the redirected radarsignals are received by the receiver 222 of the radar system 103 forprocessing.

With reference to FIG. 3, a representative transmitting channel 220 isdepicted along with a respective receiving channel 222 of the radarsystem of FIG. 3, in accordance with an exemplary embodiment. Asdepicted in FIG. 3, the transmitting channel 220 includes a signalgenerator 302, a filter 304, an amplifier 306, and an antenna 308. Alsoas depicted in FIG. 3, the receiving channel 222 includes an antenna310, an amplifier 312, a mixer 314, and a sampler/digitizer 316. Incertain embodiments the antennas 308, 310 may comprise a single antenna,while in other embodiments the antennas 308, 310 may comprise separateantennas. Similarly, in certain embodiments the amplifiers 306, 312 maycomprise a single amplifier, while in other embodiments the amplifiers306, 312 may comprise separate amplifiers.

The radar system 103 generates the transmittal radar signals via thesignal generator 302. The transmittal radar signals are filtered via thefilter(s) 304, amplified via the amplifier 306, and transmitted from theradar system 103 (and from the vehicle 10 to which the radar system 103belongs, also referred to herein as the “host vehicle”) via the antenna308. The transmitting radar signals subsequently contact other vehiclesand/or other objects on or alongside the road on which the host vehicle10 is travelling. After contacting the objects, the radar signals arereflected, and travel from the other vehicles and/or other objects invarious directions, including some signals returning toward the hostvehicle 10. The radar signals returning to the host vehicle 10 (alsoreferred to herein as received radar signals) are received by theantenna 310, amplified by the amplifier 312, mixed by the mixer 314, anddigitized by the sampler/digitizer 316.

Returning to FIG. 2, the radar system 103 also includes, among otherpossible features, the memory 224 and the processing unit 226. Thememory 224 stores information received by the receiver 222 and/or theprocessing unit 226. In certain embodiments, such functions may beperformed, in whole or in part, by a memory 242 of the computer system232 (discussed further below).

The processing unit 226 processes the information obtained by thereceivers 222 for classification of objects using radar signalspectrogram data in combination with one or more computer vision models.The processing unit 226 of the illustrated embodiment is capable ofexecuting one or more programs (i.e., running software) to performvarious tasks instructions encoded in the program(s). The processingunit 226 may include one or more microprocessors, microcontrollers,application specific integrated circuits (ASICs), or other suitabledevice as realized by those skilled in the art, such as, by way ofexample, electronic control component, processing logic, and/orprocessor device, individually or in any combination, including withoutlimitation: application specific integrated circuit (ASIC), anelectronic circuit, a processor (shared, dedicated, or group) and memorythat executes one or more software or firmware programs, a combinationallogic circuit, and/or other suitable components that provide thedescribed functionality.

In certain embodiments, the radar system 103 may include multiplememories 224 and/or processing units 226, working together orseparately, as is also realized by those skilled in the art. Inaddition, it is noted that in certain embodiments, the functions of thememory 224, and/or the processing unit 226 may be performed in whole orin part by one or more other memories, interfaces, and/or processorsdisposed outside the radar system 103, such as the memory 242 and theprocessor 240 of the controller 104 described further below.

As depicted in FIG. 2, the controller 104 is coupled to the radar system103. Similar to the discussion above, in certain embodiments thecontroller 104 may be disposed in whole or in part within or as part ofthe radar system 103. In addition, in certain embodiments, thecontroller 104 is also coupled to one or more other vehicle systems(such as the electronic control system 118 of FIG. 1). The controller104 receives and processes the information sensed or determined from theradar system 103, provides detection, classification, and tracking ofobjects, and implements appropriate vehicle actions based on thisinformation. The controller 104 generally performs these functions inaccordance with the method 400 discussed further below in connectionwith FIGS. 4-8.

As depicted in FIG. 2, the controller 104 comprises a computer system232. In certain embodiments, the controller 104 may also include theradar system 103, one or more components thereof, and/or one or moreother systems. In addition, it will be appreciated that the controller104 may otherwise differ from the embodiment depicted in FIG. 2. Forexample, the controller 104 may be coupled to or may otherwise utilizeone or more remote computer systems and/or other control systems, suchas the electronic control system 118 of FIG. 1.

As depicted in FIG. 2, the computer system 232 includes a processor 240,a memory 242, an interface 244, a storage device 246, and a bus 248. Theprocessor 240 performs the computation and control functions of thecontroller 104, and may comprise any type of processor or multipleprocessors, single integrated circuits such as a microprocessor, or anysuitable number of integrated circuit devices and/or circuit boardsworking in cooperation to accomplish the functions of a processing unit.In one embodiment, the processor 240 classifies objects using radarsignal spectrogram data in combination with one or more computer visionmodels. During operation, the processor 240 executes one or moreprograms 250 contained within the memory 242 and, as such, controls thegeneral operation of the controller 104 and the computer system 232,generally in executing the processes described herein, such as those ofthe method 400 described further below in connection with FIGS. 4-8.

The memory 242 can be any type of suitable memory. This would includethe various types of dynamic random access memory (DRAM) such as SDRAM,the various types of static RAM (SRAM), and the various types ofnon-volatile memory (PROM, EPROM, and flash). In certain examples, thememory 242 is located on and/or co-located on the same computer chip asthe processor 240. In the depicted embodiment, the memory 242 stores theabove-referenced program 250 along with one or more stored values 252(such as, by way of example, information from the received radar signalsand the spectrograms therefrom).

The bus 248 serves to transmit programs, data, status and otherinformation or signals between the various components of the computersystem 232. The interface 244 allows communication to the computersystem 232, for example from a system driver and/or another computersystem, and can be implemented using any suitable method and apparatus.The interface 244 can include one or more network interfaces tocommunicate with other systems or components. In one embodiment, theinterface 244 includes a transceiver. The interface 244 may also includeone or more network interfaces to communicate with technicians, and/orone or more storage interfaces to connect to storage apparatuses, suchas the storage device 246.

The storage device 246 can be any suitable type of storage apparatus,including direct access storage devices such as hard disk drives, flashsystems, floppy disk drives and optical disk drives. In one exemplaryembodiment, the storage device 246 comprises a program product fromwhich memory 242 can receive a program 250 that executes one or moreembodiments of one or more processes of the present disclosure, such asthe method 400 (and any sub-processes thereof) described further belowin connection with FIGS. 4-8. In another exemplary embodiment, theprogram product may be directly stored in and/or otherwise accessed bythe memory 242 and/or a disk (e.g., disk 254), such as that referencedbelow.

The bus 248 can be any suitable physical or logical means of connectingcomputer systems and components. This includes, but is not limited to,direct hard-wired connections, fiber optics, infrared and wireless bustechnologies. During operation, the program 250 is stored in the memory242 and executed by the processor 240.

It will be appreciated that while this exemplary embodiment is describedin the context of a fully functioning computer system, those skilled inthe art will recognize that the mechanisms of the present disclosure arecapable of being distributed as a program product with one or more typesof non-transitory computer-readable signal bearing media used to storethe program and the instructions thereof and carry out the distributionthereof, such as a non-transitory computer readable medium bearing theprogram and containing computer instructions stored therein for causinga computer processor (such as the processor 240) to perform and executethe program. Such a program product may take a variety of forms, and thepresent disclosure applies equally regardless of the particular type ofcomputer-readable signal bearing media used to carry out thedistribution. Examples of signal bearing media include: recordable mediasuch as floppy disks, hard drives, memory cards and optical disks, andtransmission media such as digital and analog communication links. Itwill similarly be appreciated that the computer system 232 may alsootherwise differ from the embodiment depicted in FIG. 2, for example inthat the computer system 232 may be coupled to or may otherwise utilizeone or more remote computer systems and/or other control systems.

FIGS. 4A-4C are flowcharts of a method 400 for implementing a radarsystem of a vehicle, in accordance with an exemplary embodiment. Themethod 400 can be implemented in connection with the vehicle 10 of FIG.1 and the radar control system 12 of FIGS. 1-3, in accordance with anexemplary embodiment.

As depicted in FIG. 4A, in one embodiment the method 400 includes atraining method 401 and an implementation method 402. In one embodiment,the training method 401 is performed offline, for example duringmanufacture of the vehicle and/or otherwise prior to the current vehicledrive or ignition cycle, and includes training of a model forclassification of objects. Also in one embodiment, the implementationmethod 401 is performed online, for example during a current vehicledrive or ignition cycle, and implements the model in classification ofobjects encountered during the current vehicle drive or ignition cycle.The training method 401 will be discussed below in connection with FIG.4B, and the implementation method 402 will be discussed further below inconnection with FIG. 4C, in accordance with exemplary embodiments.

With reference to FIG. 4B, in various embodiments the training method401 can be scheduled to run at 403 based on predetermined “offline”events (for example during manufacture of the vehicle, prior to sale ofthe vehicle to the customer/owner, and/or otherwise prior to the currentvehicle drive or ignition cycle). As depicted in FIG. 4B, in oneembodiment the training method 401 includes transmitting a firstplurality of radar signals of a first waveform at 404. The radar signalsare, in one example, transmitted via the transmitting channel 220 of theradar system 103 of the vehicle 10 of FIG. 1 while the vehicle 10 isdriving in a road.

After the radar signals are reflected from objects on or around theroad, return radar signals are received by the radar system 103 at 406of FIG. 4B. In one example, the received radar signals are received viathe receiving channel 222 of the radar system 103 of the vehicle 10 (asreferenced in FIGS. 1-3) after deflection from one or more objects (e.g.other vehicles, pedestrians, trees, rocks, debris, road characteristics,and so on) on the road or otherwise in proximity to the vehicle 10. Theobject is detected at 427 based upon the radar received radar signals.

A first region is selected with respect to the return radar signals at408. In one embodiment, the first region comprises a three dimensionalspatial region in proximity to an object (hereafter referred to as “theobject”) from which the received radar signals had been reflected withrespect to 405. In one such embodiment, the first region comprises acube-shaped region proximate a portion of the object (such that multiplesuch cube-shaped regions would provide a more complete representationfor the object). In one embodiment, the first region is selected by aprocessor, such as the processing unit 226 and/or the processor 240 ofFIG. 2. In one embodiment, for each detected target, its surrounding isrepresented by spatial cells (cubes), and the information form each cellis treated as a separate data stream. Also in one embodiment, the sizeof the cubes is determined by the radar resolution inRange-Azimuth-Elevation.

Information pertaining to the first region is analyzed at 410. In oneembodiment, the information includes a correspondence or plot of thefrequency contents (i.e., the spectrogram) of the received radar signalsover time. The resulting analyzed data (including the correspondence ofthe frequency over time) is represented in a spectrogram generated at412 for the first region. The analysis of the information and generationof the spectrogram are performed by a processor, such as the processingunit 226 and/or the processor 240 of FIG. 2.

A determination is made at 414 as to whether analysis is required forany additional regions. In one embodiment, the determination of 414 iswhether there any remaining spatial regions proximate the object forwhich a spectrogram has not yet been generated. In one embodiment, thisdetermination is made by a processor, such as the processing unit 226and/or the processor 240 of FIG. 2.

If it is determined that analysis is required for one or more additionalregions requiring analysis, a new region is selected at 416. In oneembodiment, at 416 a new three dimensional spatial region is selected,similar to the first region of 408, but at a slightly different locationin proximity to the object. In one embodiment, the new region isselected by a processor, such as the processing unit 226 and/or theprocessor 240 of FIG. 2. The method proceeds to 410, and 410-416 repeatin this manner until there are no additional regions required foranalysis (as determined at 414), at which point the method proceeds to418, described below.

After 403-416 are repeated for a number of different vehicles (forexample, test vehicles prior to sale to the public), the information (ordata) from the various different spectrograms for the object for thevarious vehicles are combined or aggregated to create a collection ordatabase at 418. This is preferably performed by a processor, such asthe processing unit 226 and/or the processor 240 of FIG. 2.

At 420, features are extracted from the spectrograms in the database of418. In one embodiment, the features are extracted in conjunction with acomputer vision model that is trained during the training method 401. Inone embodiment, the features are classified at 422 as part of thetraining of the computer vision model during the training method 401. Asdepicted in FIG. 4B, in one embodiment the feature extraction of 420 andthe classification of 422 may be referred to as sub-process 423 (forease of reference with respect to FIGS. 5-8 below). In one embodiment,the sub-process 423 is performed by a processor, such as the processingunit 226 and/or the processor 240 of FIG. 2, in training a computervision model that is then stored in memory, such as the memory 224and/or the memory 244 of FIG. 2.

With reference to FIGS. 5-8, exemplary embodiments of the sub-process423 of FIG. 4B are discussed below, including (i) one embodiment inwhich feature vectors and vector quantization are used with a bag ofwords model (as depicted in FIGS. 5 and 6); and (ii) one embodiment inwhich binary vectors are utilized by comparing energy intensities of thespectrogram data in conjunction with a regression tree model (asdepicted in FIGS. 7 and 8). Unless otherwise noted, the method of theseembodiments is performed by a processor, such as the processing unit 226and/or the processor 240 of FIG. 2.

With reference to FIGS. 5 and 6, in one embodiment (in which a bag ofwords model is used) each region is detected at 502. As depicted in FIG.6, in one embodiment, the selection corresponds to various regions 602of each spectrogram 604 (for example, in which the x-axis 601 denotestime, in seconds, and the y-axis 603 denotes frequency, in Hz). In oneembodiment, each region 602 represents a picture pertaining to theobject, and the various pictures of the different regions 602collectively comprise the image of the object from the spectrogram.

Features are extracted for each region at 504. In certain embodiments,the extracted features are extracted in the form of a matrix 606, andinclude a scale-invariant feature transform (SIFT) descriptor and/orwavelet coefficients for each region 602 of the spectrogram 604. In oneembodiment, the features are extracted and embedded into a matrix withone raw per feature vector describing the patch. In various embodiments,any one of a number of different filters may be used (e.g., steering,Gabor, as well as nonlinear filters)

A vector quantization is performed at 506 for the extracted features. Asdepicted in FIG. 6, in one embodiment, each of the extracted featureshas a corresponding relationship 608 with a corresponding cluster 610 ofdata points for a vector quantization in accordance with a threedimensional plot. In one embodiment, a k-means vector quantization isused. In other embodiments, any number of other clustering methods maybe used. Also as depicted in FIG. 6, the plurality of clusters 610 maybe identified as a first cluster 611, a second cluster 612, a thirdcluster 613, a fourth cluster 614, and so on, based on common featuresand characteristics.

The vector quantization output is used to build or train a computervision model through classification of objects at 508. In oneembodiment, the vector quantization output is used to build a bag ofwords representation, e.g. visual words histograms that are further usedas inputs for training the classifiers. In one embodiment, the featuresare represented as a plurality of histograms of clusters by incorporatedthe vector quantization used to build a bag of words computer visionmodel based on syntactic representations of the features. Also in oneembodiment, the classifiers may be trained separately from histogramsand attached to them object class information. With reference to FIG. 6,in one embodiment, the output from the bag of words computer visionmodel takes the form of various clustered histograms 620, wherein theimages are represented as histograms of visual words. The visual wordscorrespond to clusters of feature vectors to which patches belong (in asoft or hard way). Image histograms and image labels are used to trainthe classifiers for the different types of objects (e.g. another car, apedestrian, a bicycle, a wall, a tree, and so on) for classification. Inone embodiment, once the model is built and the classifiers trained,they are stored in memory (such as the memory 224 and/or the memory 244of FIG. 2) for subsequent retrieval from memory during theimplementation method 402 of FIG. 4C (described further below) toclassify objects encountered during a current vehicle or ignition cyclein which the vehicle is being driven on a road.

With reference to FIGS. 7 and 8, in another embodiment (in which theenergy intensities of the spectrogram data are compared to generatebinary vectors associated with a regression tree model), samples of eachregion are performed at 702. As depicted in FIG. 8, in one embodiment,the samples are taken with respect to multiple regions 801, 802 of reachspectrogram 804 (for example, in which the x-axis 805 denotes time, inseconds, and the y-axis 807 denotes frequency, in Hz). In oneembodiment, the regions 801, 802 are selected randomly, two at a time,through various multiple iterations. Also in one embodiment, each region801, 802 represents a picture pertaining to the object, and the variouspictures of the different regions 801, 802, . . . (e.g., includingnon-selected regions) collectively comprise the image of the object fromthe spectrogram. In one embodiment, these features can be efficientlycalculated using integral images.

At 704, image energy intensities are compared for the regions 801, 802of each respective pair of regions from the spectrogram 804 to generatebinary vectors. In one embodiment depicted in FIG. 8, 704 is performedthrough multiple comparisons 805 of various pairs of the randomlyselected regions 801, 802 (two such comparisons are shown in FIG. 8 forillustrative purposes, although any number of such comparisons 805 maybe performed). Also in one embodiment, the resulting comparisons areused at 704 to generate a binary vector 806, as depicted in FIG. 8. Inone such example, if the energy intensity of the first randomly selectedimage 801 is greater than or equal to the energy intensity of the secondrandomly selected image 802, then a binary digit of zero (0) is providedfor the binary vector 806 for this comparison. Also in one such example,if the energy intensity of the first randomly selected image 801 is lessthan the energy intensity of the second randomly selected image 802,then a binary digit of one (1) is provided for the binary vector 806 forthis comparison. In one embodiment, 702 and 704 are repeated for variousvehicles (e.g. in development and/or otherwise being sold to consumers),resulting in numerous binary vectors 806 for the various vehicles.

At 706, the binary vectors 806 are used to train the computer visionmodel through object classification. In one embodiment, the binaryvectors 806 are associated with known object information (e.g. the typesof objects) during the training of the model using sums of squares for amodified fern regression tree model, so that the binary vectors (or theassociated sums of squares) can be associated with known types of knownobjects (e.g., another car, a pedestrian, a bicycle, a wall, a tree, andso on) from the training data, to thereby train regression model. Alsoas shown in FIG. 8, graphical depictions 830 may be utilized for theobject classification and training of the regression model in trainingthe model to recognize different features 840 (for example, a firstfeature 841, a second feature 842, a third feature 843, and so on)and/or patterns of features as being associated with specific types ofobjects (for example, a first feature 841, a second feature 842, a thirdfeature 843, and so on of the data and the objects). Once the regressionmodel is trained, the model is stored in memory (such as the memory 224and/or the memory 244 of FIG. 2) at 424, so that the model cansubsequently be retrieved from memory during the implementation method402 of FIG. 4C (described further below) to classify objects encounteredduring a current vehicle or ignition cycle in which the vehicle is beingdriven on a road.

With reference to FIG. 4C, in various embodiments the implementationmethod 402 can be scheduled to run at 453 based on predetermined“online” events, and/or can run continually during operation of thevehicle 10 (e.g. a current vehicle drive or ignition cycle, after thesale of the vehicle to a customer, as the vehicle is being driven by thecustomer or owner on a roadway). As depicted in FIG. 4C, in oneembodiment the implementation method 402 includes transmitting a firstplurality of radar signals of a first waveform at 454. The radar signalsare, in one example, transmitted via the transmitting channel 220 of theradar system 103 of the vehicle 10 of FIG. 1 while the vehicle 10 isdriving in a road.

After the radar signals are reflected from objects on or around theroad, return radar signals are received by the radar system 103 at 456of FIG. 4C. In one example, the received radar signals are received viathe receiving channel 222 of the radar system 103 of the vehicle 10 (asreferenced in FIGS. 1-3) after deflection from one or more objects (e.g.other vehicles, pedestrians, trees, rocks, debris, road characteristics,and so on) on the road or otherwise in proximity to the vehicle 10. Theobject is detected at 407 based upon the radar received radar signals.

A first region is selected with respect to the return radar signals at458. In one embodiment, the first region comprises a three dimensionalspatial region in proximity to an object (hereafter referred to as “theobject”) from which the received radar signals had been reflected withrespect to 405. In one such embodiment, the first region comprises acube-shaped region proximate a portion of the object (such that multiplesuch cube-shaped regions would provide a more complete representationfor the object). In one embodiment, the first region is selected by aprocessor, such as the processing unit 226 and/or the processor 240 ofFIG. 2.

Information pertaining to the first region is analyzed at 460. In oneembodiment, the information includes a correspondence or plot of thefrequency of the received radar signals over time. The resultinganalyzed data (including the correspondence of the frequency over time)is represented in a spectrogram generated at 462 for the first region.The analysis of the information and generation of the spectrogram areperformed by a processor, such as the processing unit 226 and/or theprocessor 240 of FIG. 2.

A determination is made at 464 as to whether analysis is required forany additional regions. In one embodiment, the determination of 464 iswhether there any remaining spatial regions proximate the object forwhich a spectrogram has not yet been generated. In one embodiment, thisdetermination is made by a processor, such as the processing unit 226and/or the processor 240 of FIG. 2.

If it is determined that analysis is required for one or more additionalregions requiring analysis, a new region is selected at 466. In oneembodiment, at 466 a new three dimensional spatial region is selected,similar to the first region of 458, but at a slightly different locationin proximity to the object. In one embodiment, the new region isselected by a processor, such as the processing unit 226 and/or theprocessor 240 of FIG. 2. The method proceeds to 460, and 460-466 repeatin this manner until there are no additional regions required foranalysis (as determined at 464), at which point the method proceeds to468, described below.

At 470, features are extracted from the spectrograms for the vehiclethat have been collected during the current vehicle drive or ignitioncycle for the vehicle. In one embodiment, the features are extracted inconjunction with a computer vision model that had previously beentraining during the training method 401 discussed above. In oneembodiment, the features are classified at 472 as part of the computervision model that was previously trained during the training method 401.As depicted in FIG. 4C, in one embodiment the feature extraction of 470and the classification of 472 may be referred to as sub-process 473. Inone embodiment, the sub-process 473 is performed by a processor, such asthe processing unit 226 and/or the processor 240 of FIG. 2, using apreviously-trained computer vision model that is retrieved from memory,such as the memory 224 and/or the memory 244 of FIG. 2.

In one embodiment of 470 (and 473), the feature extraction andclassification is performed in conjunction with the bag of words modeldescribed above with respect to FIGS. 5 and 6. Specifically, in thisembodiment, during the implementation method 402 using the bag of wordsmodel, the region selection of 502 and the feature extraction of 504 areperformed as described above in connection with FIGS. 5 and 6 for thetraining method 401. However, during the implementation method 402, theclassification of 472 is performed by comparing the feature vectors forthe current vehicle drive or ignition cycle with known feature vectorsfrom the stored model from the training method 401 that are known tocorrespond with specific types of objects (e.g., another car, apedestrian, a bicycle, a wall, a tree, and so on).

In another embodiment of 470 (and 473), the feature extraction andclassification is performed in conjunction with the regression modeldescribed above with respect to FIGS. 7 and 8. Specifically, in thisembodiment, during the implementation method 402 using the regressionmodel, the region sampling of 702 and the intensity comparisons/creationof the binary vector of 704 are performed in connection with themodified regression tree model as described above in connection withFIGS. 7 and 8 for the training method 401. However, during theimplementation method 402, the classification of 472 is performed bycomparing the binary vectors for the current vehicle drive or ignitioncycle with known binary vectors from the stored model from the trainingmethod 401 that are known to correspond with specific types of objects(e.g., another car, a pedestrian, a bicycle, a wall, a tree, and so on).In one embodiment, the probabilities that particular binary vectorsbelong to different object classes are learned during the trainingstage. Also in one embodiment, subsequently, during the testing stage,new binary vectors are classified based on the maximal likelihoodcriteria (e.g., maximal class probability).

The objects may be tracked over time at 474, for example by trackingchanges in movement and/or position of the objects using the receivedradar signals and the classification. Vehicle actions may be initiatedas appropriate at 476 based upon the classification and/or tracking. Inone example, if an object is classified as an object of concern (e.g., apedestrian, bicycle, other vehicle, and/or other large object) and thedistance between the vehicle 10 and the tracked object is less than apredetermined threshold (or an estimated time of contact between thevehicle 10 and the tracked object) under their current respectivetrajectories is less than a predetermined threshold), then an alert(e.g., a visual or audio alert to the driver) may be provided and/or anautomatic vehicle control action (e.g., automatic braking and/orautomatic steering) may be initiated, for example by a processoroutputting one or more control signals for the steering system 150and/or the braking system 160 of FIG. 1. In various embodiments, themethod 400 may terminate at 478 when the action is complete, or whenfurther use of the radar system and/or the method 400 is no longerrequired (e.g. when the ignition is turned off and/or the currentvehicle drive and/or ignition cycle terminates).

Methods and systems are provided herein for classifying objects forradar systems of vehicles. The disclosed methods and systems provide forthe classification of objects using radar signal spectrogram data incombination with one or more computer vision models.

It will be appreciated that the disclosed methods, systems, and vehiclesmay vary from those depicted in the Figures and described herein. Forexample, the vehicle 10, the radar control system 12, the radar system103, the controller 104, and/or various components thereof may vary fromthat depicted in FIGS. 1-3 and described in connection therewith. Inaddition, it will be appreciated that certain steps of the method 400may vary from those depicted in FIGS. 4-8 and/or described above inconnection therewith. It will similarly be appreciated that certainsteps of the method described above may occur simultaneously or in adifferent order than that depicted in FIGS. 4-8 and/or described abovein connection therewith.

While at least one exemplary embodiment has been presented in theforegoing detailed description, it should be appreciated that a vastnumber of variations exist. It should also be appreciated that theexemplary embodiment or exemplary embodiments are only examples, and arenot intended to limit the scope, applicability, or configuration of thedisclosure in any way. Rather, the foregoing detailed description willprovide those skilled in the art with a convenient road map forimplementing the exemplary embodiment or exemplary embodiments. Itshould be understood that various changes can be made in the functionand arrangement of elements without departing from the scope of theappended claims and the legal equivalents thereof.

What is claimed is:
 1. A method for object classification for a radarsystem of a vehicle, the method comprising the steps of: obtainingspectrogram data from a plurality of spectrograms pertaining to anobject proximate the vehicle based on received radar signals for theradar system; aggregating the spectrogram data from each of theplurality of spectrograms into a computer vision model; and classifyingthe object based on the aggregation of the spectrogram data from each ofthe plurality of spectrograms into the computer vision model.
 2. Themethod of claim 1, further comprising: identifying a plurality of threedimensional spatial regions proximate the object, wherein the obtainingspectrogram data comprises obtaining spectrogram data from the pluralityof spectrograms, each of the plurality of spectrograms pertaining to acorresponding one of the plurality of three dimensional spatial regions.3. The method of claim 1, wherein the aggregating the spectrogram datacomprises aggregating the spectrogram data from each of the plurality ofspectrograms into a clustering computer vision model.
 4. The method ofclaim 3, wherein the aggregating the spectrogram data comprises:extracting features from each of the plurality of spectrograms;performing a vector quantization of the features; and incorporating thevector quantization into the clustering computer vision model.
 5. Themethod of claim 4, wherein the incorporating the vector quantizationcomprises: representing the features as a plurality of histograms ofclusters using the vector quantization and a bag of words computervision model.
 6. The method of claim 1, wherein the aggregating thespectrogram data comprises comparing energy intensities of thespectrogram data from the plurality of spectrograms via calculated sumsof values from each of the plurality of spectrograms using the computervision model.
 7. The method of claim 6, wherein the aggregating isperformed using a modified regression tree computer vision model.
 8. Aradar control system for a vehicle, the radar control system comprising:a transmitter configured to transmit radar signals; a receiverconfigured to receive return radar signals after the transmitted radarsignals are deflected from an object proximate the vehicle; and aprocessor coupled the receiver and configured to: obtain spectrogramdata from a plurality of spectrograms pertaining to the object based onthe received radar signals; aggregate the spectrogram data from each ofthe plurality of spectrograms into a computer vision model; and classifythe object based on the aggregation of the spectrogram data from each ofthe plurality of spectrograms into the computer vision model.
 9. Theradar control system of claim 8, wherein the processor is furtherconfigured to: identify a plurality of three dimensional spatial regionsproximate the object, wherein each of the plurality of spectrogramspertains to a corresponding one of the plurality of three dimensionalspatial regions.
 10. The radar control system of claim 8, wherein theprocessor is further configured to aggregate the spectrogram data fromeach of the plurality of spectrograms into a clustering computer visionmodel.
 11. The radar control system of claim 10, wherein the processoris further configured to: extract features from each of the plurality ofspectrograms; perform a vector quantization of the features; andincorporate the vector quantization into the clustering computer visionmodel.
 12. The radar control system of claim 11, wherein the processoris further configured to represent the features as a plurality ofhistograms of clusters using the vector quantization and a bag of wordscomputer vision model.
 13. The radar control system of claim 8, whereinthe processor is further configured to compare energy intensities of thespectrogram data from the plurality of spectrograms via calculated sumsof values from each of the plurality of spectrograms using the computervision model.
 14. The radar control system of claim 13, wherein thecomputer vision model comprises a modified regression tree computervision model.
 15. A computer system for object classification for aradar system of a vehicle, the computer system comprising: anon-transitory, computer readable storage medium storing a program, theprogram configured to at least facilitate: obtaining spectrogram datafrom a plurality of spectrograms pertaining to the object based onreceived radar signals for the radar system; aggregating the spectrogramdata from each of the plurality of spectrograms into a computer visionmodel; and classifying the object based on the aggregation of thespectrogram data from each of the plurality of spectrograms into thecomputer vision model.
 16. The computer system of claim 15, wherein theprogram is further configured to at least facilitate: identifying aplurality of three dimensional spatial regions proximate the object,wherein each of the plurality of spectrograms pertains to acorresponding one of the plurality of three dimensional spatial regions.17. The computer system of claim 15, wherein the program is furtherconfigured to at least facilitate aggregating the spectrogram data fromeach of the plurality of spectrograms into a clustering computer visionmodel.
 18. The computer system of claim 17, wherein the program isfurther configured to at least facilitate: extracting features from eachof the plurality of spectrograms; performing a vector quantization ofthe features; and incorporating the vector quantization into theclustering computer vision model.
 19. The computer system of claim 18,wherein the program is further configured to at least facilitaterepresenting the features as a plurality of histograms of clusters usingthe vector quantization and a bag of words computer vision model. 20.The computer system of claim 15, wherein the program is furtherconfigured to at least facilitate comparing energy intensities of thespectrogram data from the plurality of spectrograms via calculated sumsof values from each of the plurality of spectrograms using a modifiedregression tree computer vision model.